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The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )S Responsive to communication(s) filed on 29 June 2001 . 
2a)D This action is FINAL. 2b)^ This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 
Disposition of Claims 

4) IEI Claim(s) 1-18 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) ^ Claim(s) 1-18 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) Q The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

11) D The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§ 119 and 120 

13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)D All b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. Q Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 
Attachment(s) 

1) M Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-413) Paper No(s). . 

2) D Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) □ Notice of Informal Patent Application (PTO-152) 

3) O Information Disclosure Statement(s) (PTO-1449) Paper No(s) 2A* 6) O Other: 



U.S. Patent and Trademark Office 

PTO-326 (Rev. 04-01 ) Office Action Summary Part of Paper No. 5 
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DETAILED ACTION 



Claim Rejections - 35 USC §103 



1. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as 
set forth in section 102 of this title, if the differences between the subject matter sought to be patented 
and the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 



2. Claims 1-18 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Inagaki (5,999,214) in view of Pavlovic et al. "Integration of audio/visual information 
for use in human-computer intelligent interaction", Image processing, 1997 Proceedings 
IEEE pages 121. 

With regard to claim 1 Inagaki teaches a video display device comprising: a 
display configured to display a primary image and a picture-in-picture image (PIP) 
overlaying the primary image (figure 1 1, items 13 and 17); a processor operatively 
coupled to the display and configured to receive a first video data stream for the primary 
image, to receive a second video data stream for the PIP (figure 11, items 22 and 16), 

Inagaki does not teach, "and to change a PIP display characteristic in response to 
a received audio indication and a related gesture from a user". Inagaki instead only uses 
the audio indication from a user and does not depend on a "related gesture from a user" 
(figure 11 "VOICE DIRECTION DETECTION UNIT", column 3, lines 31-33, column 
10, lines 16-25). 
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However, Pavlovic et al. does teach integrating both audio and visual gesture 
information to manipulate virtual objects displayed on the projection screen ( page 123 
3. EXPERIMENTAL RESULTS section ). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to use a "received audio indication and a related gesture from a 
user" as taught by Pavlovic in the apparatus of Inagaki, because of the motivation 
provided by Pavlovic "Psychological studies, for example, show that people prefer to 
use hand gestures in combination with speech in a virtual environment, since they allow 
the user to interact without special training or special apparatus". 

With regard to claim 2 the combination of Inagaki and Pavlovic teaches the 
video display device of claim 1, wherein the PIP display characteristic is at least one of a 
position of the PIP on the display and a display size of the PIP (See Inagaki figure 8a). 

With regard to claim 3 the combination of Inagaki and Pavlovic teaches the 
video display device of claim 1, comprising: a microphone for receiving the audio 
indication from the user; and a camera for acquiring an image of the user containing the 
related gesture (See Inagaki figure 1 1). 

With regard to claim 4 the combination of Inagaki and Pavlovic teaches the 
video display device of claim 1 wherein the processor is configured to analyze audio 
information received from the user to identify when a PIP related audio indication is 
intended by the user (See Inagaki figure 8a and 8b). 
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With regard to claim 5 the combination of Inagaki and Pavlovic teaches the 
video display device of claim 1, wherein the processor is configured to analyze image 
information received from the user after the audio indication is received to identify the 
change in the PIP display characteristic that is expressed by the received gesture (See 
Inagaki figure 8a and 8b and Pavlovic et al figures 6-8). 

With regard to claim 6 the combination of Inagaki and Pavlovic teaches the 
video display device of claim 5, wherein the image information is contained in a 
sequence of images and wherein the processor is configured to analyze the sequence of 
images to determine the received gesture (since a gesture can be a motion which would 
require a sequence of images to detect this feature is inherent to the system of Inagaki 
and Pavlovic). 

With regard to claim 7 the combination of Inagaki and Pavlovic teaches the 
video display device of claim 1, wherein the image information is contained in a 
sequence of images and wherein the processor is configured to determine the received 
gesture by analyzing the sequence of images and determining a trajectory of a hand of 
the user (since a gesture can be a motion which would require a sequence of images to 
detect this feature is inherent to the system of Inagaki and Pavlovic). 

With regard to claim 8 the combination of Inagaki and Pavlovic teaches the 
video display device of claim 1, wherein the processor is configured to determine the 
received gesture by analyzing an image of the user and determining a posture of a hand 
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of the user (since a gesture can be a posture of a hand this feature is inherent to the 
system of Inagaki and Pavlovic). 

With regard to claim 9 the combination of Inagaki and Pavlovic suggest the 
video display device of claim 1, wherein the video display device is a television (since 
Pavlovic shows a projection screen in figure 6 and since it is also well-known in the 
prior art that televisions use projection screens one would be motivated to have a 
projection screen with a dual use such as conference and watching the game . 

With regard to claim 9 the combination of Inagaki and Pavlovic teaches the 
video display device of claim 1, wherein the image is a sequence of images of the user 
containing the user gesture, the video display device comprising a camera for acquiring 
the sequence of images of the user (see Inagaki figure 1 1, item 2). 

With regard to claims 1 1-14 the combination of Inagaki and Pavlovic was shown 
with regards to apparatus claims 1-10 to be obvious and therefore the method claims 11- 
14 which corresponds to the apparatus is also obvious. 

With regard to claims 15-18 the combination of Inagaki and Pavlovic was shown 
above to read on most of these limitation in claims 1-10 in addition to summarize a 
feature directed towards a program stored implementing this process is inherent to the 
automatic computer system taught by the combination of Inagaki and Pavlovic. 
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Conclusion 

3. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Paul Bell whose telephone number is (703) 306-3019. If 
attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, 
Steven Saras, can be reached at (703) 305-9720. 

Any response to this action should be mailed to: 

Commissioner of Patents and Trademarks 
Washington, D.C. 20231 

or faxed to: (703) 872-9314 

Hand-delivered responses should be brought to Crystal Park n, 2121 Crystal Drive, 
Arlington, VA, Sixth Floor (Receptionist). Any inquiry of a general nature or relating to 
the status of this application or proceeding should be directed to the Technology Center 
2600 Customer Service Office whose telephone number is (703) 306-0377. 

Paul Bell V 
Art unit 2675 
17 June 2003 
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